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Abstract 

This paper characterizes the sohition to a finite horizon min-max 
optimal control problem where the system is linear and discrete-time 
with control and state constraints, and the cost quadratic; the dis- 
turbance is negatively costed, as in the standard i?oo problem, and 
is constrained. The cost is minimized over control policies and maxi- 
mized over disturbance sequences so that the solution yields a feedback 
control. It is shown that the value function is piecewise quadratic and 
the optimal control policy piecewise affine, being quadratic and afhne, 
respectively, in polytopes that partition the domain of the value func- 
tion. 

Keywords: min-max, constrained, Hoo, parametric optimization, optimal control. 

1 Introduction 

Characterizations of solutions to constrained optimal control problems ap- 
peared in the papers [1-4] that deal with the constrained linear-quadratic 
problem, in the papers [4-7] and thesis [8] that deal with hybrid or piecewise 
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affine systems, and in papers that deal with min-max optimal control prob- 
lems [9-13]. In these papers it is shown that the value function is piecewise 
affine or piecewise quadratic (depending on the nature of the cost function 
in the optimal control problem) and the control law is piecewise affine, being 
quadratic or affine in polytopes that constitute a polytopic partition of the 
domain of the value function. When disturbances are present, it is necessary 
to compute the solution sequentially using dynamic programming as in [10]. 
In this paper, which is motivated by recent research on H^o model predic- 
tive control [14-21], we obtain an explicit characterization of the solution 
to a constrained, min-max optimal control problem and consider here the 
choice of terminal cost and constraint set to ensure stability of the closed 
loop system with receding horizon control. The term Hao is used somewhat 
loosely since we consider the min-max problem with fixed 7. We consider, 
therefore, the problem of controlling a linear, discrete-time system described 
by 

x'^ = Ax + Bu + Gw, y = Cx + Du (1.1) 

where x G is the state, u £ M"^ the control and w € IW an additive 
disturbance (the 'adversary'); x"*" is the successor state and y € is the 
costed output. We frequently write the system dynamics in (|l.lj) in the form 

x^ = /(x, u, w) 

where f{x,u,w) = Ax + Bu + Gw. The system is subject to hard control 
and state constraints 

u e t/, xeX (1.2) 

where U C JR™ is a (compact) polytope and X C iR" a polytope; each 
set contains the origin in its interior (the assumption that X is a polytope 
rather than a polyhedron^ is made for simplicity). The disturbance w is 
constrained to lie in the polytope W C IRP] W contains the origin in its 
interior. 

Let vr = {;Uo(-)) A'i(')) • • • ) A*Af-i(')} denote a control policy (sequence 
of control laws) over horizon N and let w = {wq,wi, . . . ,wn-i} denote 
a sequence of disturbances. Also, let 4>{v, x, vr, w) denote the solution of 
(|1.1|) when the initial state is x at time 0, the control policy is vr and the 
disturbance sequence is w, so that 0(i;x,7r,w) is the solution, at time i of 

Xj+i = Axi + Bni{xi) + Gwi (1.3) 

xq = X (1-4) 

polyhedron is a set described by a finite set of inequalities; a polytope is a bounded 
polyhedron. 
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The cost Vj\f{x,7r,w), if the initial state is x, the control policy tt and the 
disturbance sequence w, is 

N-l 

VNix,7r,w) = i{xi,Ui,Wi) + Vf{xN) (1-5) 

i=0 

where, for all i, xi = (/>(i; x, vr, w) and Ui = fJ.i{xi); Vy(-) is a terminal cost 
that may be chosen, together with a terminal constraint set Xf defined 
below, to ensure stability of the resultant receding horizon controller (see 
§6). The stage cost £{■) is a quadratic function, positive definite in x and u, 
and negative definite in w: 

i{x,u,w) ^ {l/2)\x\l + (l/2)|n|| - (7V2)kP (1-6) 

where 7 > 0, = z'Zz, and Q and R are positive definite. The stage cost 
may be expressed as 

l{x,u,w) ^ (l/2)|y|2 - ij^/2)\w\\ y^Hz (1.7) 

where z = {x,u) and H is a suitably chosen matrix {{x,u) should be inter- 
preted as a column vector (x' , u')' in matrix expressions). The terminal cost 
Vf{-) is a quadratic function 

Vf{x)^{l/2)\x\% (1.8) 

in which Pf is positive definite. The optimal control problem PAr(x) that 
we consider is 

PAr(x) : V;v(^)= inf max yAr(x, vr, w) (1.9) 
7renjv(x) wgw 

where W = ^ is the set of admissible disturbance sequences, and nAr(x) is 
the set of admissible policies, i.e. those policies that satisfy, for all w G W = 
, the state and control constraints (|1.2j) . and the terminal constraint 

XN^Xf. (1.10) 

Inclusion of the hard disturbance constraint w G W is necessary when state 
constraints are present since, otherwise, for any policy tt chosen by the 
controller, we can expect that there exists a disturbance sequence w that 
transgresses the state constraint. The terminal constraint set is a polytope, 
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containing the origin in its interior, that satisfies C X, ensuring satisfac- 
tion of the state constraint at time A^. Hence the set of admissible pohcies 
is 

IIn{x) = {tt I X, vr, w) € X, x, vr, w)) £ U,i = 0,1, N — 1, 

ct>{N; X, TT, w) € Xf, Vw G W} (1.11) 

Let X]\f denote the set of initial states for which a solution to ¥j\f{x) exists 
(the domain of V^i-), the controllability set), i.e. 

XN = {x\UNix)^(D}. (1.12) 

In addition to characterizing the solution to a min-max optimal control 
problem that has not previously been characterized, this paper provides an 
improvement of the transformation procedure used in [4, 7, 8] to obtain a 
parametric solution to the optimal control problem; the improvement sim- 
plifies the determination of the polytopes in which the control law and value 
function are affine and quadratic respectively and avoids unnecessary sub- 
partitioning of overlapping polytopes required in [4,8]. 

2 Dynamic Programming for Constrained Prob- 
lems 

The solution to may be obtained as follows. For all j G = 

{1,2, . . .}, let problem Fj, the partial return function Vj*{-), and the con- 
trollability set Xj be defined as in ()1.5() - (|1.9() with j replacing N; j denotes 
"time-to-go" . Then the sequences {Vj'{-), Kj{-), Xj}, where Kj(-) denotes the 
optimal control law /i^_j(-) at time i = N—j, may be calculated recursively 
as follows [10,22]: 

V.°{x) = mmmax{i{x,u,w) + V°_^{f{x,u,w)) \ f{x,u,W) C Xj^i} (2.1) 

Kj{x) = argminmax{£(x,u, lo) + V^_i{f{x,u,w)) \ 

f{x,u,W)CXj^i} (2.2) 
Xj = Xn{x\3u£U such that f{x, u, W) C Xj^i} (2.3) 

with boundary conditions 

V,'ix) = Vfix), Xo = Xf. (2.4) 
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The condition f{x,u, W) C Xj^i in H2.2|) and H2.3|l may be expressed as 

Ax + Bue Xj^i Q GW (2.5) 

where G denotes Pontryagin set difference defined by AqB = {x \ {x]®B C 
A\ (© denotes set addition). For each integer j let Zj C iR" x iR™ be defined 
by 

Zj = {{x,u) G X X [/ I f{x,u,W) C Xj} (2.6) 
so that, from (|2.3() . 

= Projjf 

Here, and in the sequel, if a set Z, say, lies in a product space iR" x 
iR™, Projj5(^ : iR" x iR™ — > iR" denotes the projection operator defined 
by Vioix^ = {x I 3u G iR™ such that (x,tt) € Z} (iR" is regarded as x- 
space). Similarly, if $ is a set in the product space iR" x BrU^ x MP, Proj^<l> 
denotes the set {z \ 3w € IRP such that {z,w) € <J*}. We can now establish 
some preliminary properties of the solution to Ptv- To analyze PAr(x) it is 
convenient to introduce the functions J^{-)-, J = 1,2, . . ., defined by 

J^{x, u) = max{^(x, u, w) + Vfifix, u, w))} (2.7) 

The recursive equations (|2.H) - (|2.3j) may therefore be rewritten as 

K°(x) = min{J°_i(x,'u) [ f{x,u,W) C X,_i} (2.8) 
ueU 

J°_i(x,n) ^ max{^(2;,M,u;) + K0_i(/(2;,u,u;))} (2.9) 

K,(x) = argmin{J°„i(x,M) | f{x,u,W) C X,_i} (2.10) 

Xj = Xn{x\3ueU St f{x, u, W) C Xj^i] (2.11) 

for j = 1, . . . , X with endpoint conditions Vo(-) = V/(-), Xq = Xj. Under 
our assumptions the sets Xj and Zj are compact. If Xq = Xj is robust 
control invariant, the sets Xj are nested (Xj ^ Xj_i for all j > 1). For each 
J, the domain of Vji') includes Xj but we are only interested in its values 
on Xj; similarly, the domain of J^{ ) includes Zj but we are only interested 
in its values on Zj. 

3 Parametric Optimization 

We seek a parametric solution to problem P7v(a^), i-e. a solution for all values 
of the parameter which, in this case, is the state x. More precisely, since we 
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employ constrained dynamic programming, we seek a parametric solution 
to problems ¥j{x) for all j E {1, • • • ,N}. First, we introduce a few useful 
definitions. 

Definition 1 For any positive integer J, Zj = {1, 2, . . . , J}; for any set X , 

denotes an index set associated with a partition of X . 

Definition 2 A set V = {Pi \ i G J}, for some index set J , is called a 
polyhedral (polytopic) partition of a closed (compact) set X if X = Ui^jPi, 
and the sets Pi, i ^ J are polyhedrons (polytopes) with non-empty interiors 
which are non-intersecting (Interior(Pj)ninterior(Pj) = for alli,j (z J,i ^ 

j)- 

Definition 3 A function V : X ^ M is said to be continuous piecewise 
quadratic on a polyhedral (polytopic) partition V = {Pi \ i € J} of X if it 
is continuous and satisfies 

V{x) = {l/2)\x\l^ + q'iX + ri, yx e Pi, i e J 

for some Qi,qi,ri, i ^ J . Similarly, a function k : X ^ U is said to he 
piecewise affine on a polyhedral partition V = {Pi \ i G JT"} of X if it is 
continuous and satisfies 

k{x) = KiX + ki, Vx € Pi, i £ J', 

for some Ki,ki, i£ J , where V has the properties specified above. 

The dynamic programming recursion (|2.8() - H2.11() requires the repeated 
solution of two prototype problems Pmin and Pmax defined next: 

Fmin(x) : y°(x) = mm{J{x,u) \ {x,u) € Z} (3.1) 

u 

Pmax(-z) : J^iz) = max{V{z,w) \w eW} (3.2) 

w 

The minimizer in Pmin(a^) and the maximizer in Pmax(-2) are defined, respec- 
tively, by 

k{x) = argminj J(2;, n) [ {x,u) G Z} (3-3) 

u 

v{z) = argmax{y(z, w) | w G W}. (3-4) 

w 

Problem Pmin(a^) is the prototype for Problem (|2.8|) with V^{x) replacing 
Vj{x), J{x,u) replacing J^_-^{x,u), and {x,u) G Z replacing the constraints 
f{x, u, W) C Xj^i {Ax + Bue Xj_i e GW) and ueU. Similarly Problem 
Pmax(-2) is the prototype for Problem (|2.9() with J^{z) replacing Jj_^{z), 
V{z,w) replacing i{x,u,w) + V^_i{f{x,u,w)), and z replacing {x,u). We 
first obtain the parametric solution of Pmin- 
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3.1 The minimization problem P, 



The solution to Pmin(2;) has properties given in Proposition ^ that has a 
simpler hypothesis than previous versions of this result. For completeness, 
continuity of the control law is also proven. 

Proposition 1 Suppose J : Z ^ ]R is a strictly convex, continuous func- 
tion and that Z is a polytope. Then, for all x ^ X = Proj^^; the solution 
hi{x) to Pmin(2;) cxists and is unique. The value function V^{-) is strictly 
convex and continuous with domain X , and the control law n{-) is continuous 
on X . 

Proof: For all x (z X , U{x) = {u \ {x, u) € Z^ is convex and compact. 
Let A := {(Ai,A2) | Ai > 0, A2 > 0, Ai + A2 = 1}. For all xi,X2 in X, all 
A = (Ai,A2) G A: 

y°(AiXi + A2X2) = min{J(Aia;i + A2X2,'u) I (Aixi + A2a;2,u) G 2} 

u 

< J{\lXl+ \2X2,\lUi + X2U2}, Ui = K{Xi), i = l,2 
= J(Ai(xi,lii) + A2(X2,U2)) 

But Xi{xi, ui) + A2(2;2, U2) € Z since Z is convex and (xj, Ui) & Z, i = 1,2. 
Since J(-) is strictly convex 

y°(AiXi + A2X2) < Ai J(2;i,'Ui) + A2J(a;2,n2) 

= XiV^ixi) + X2V^{x2) VAi, A2 G A 

where the last inequality is strict if Ai {0, 1} so that V^{-) is strictly 
convex. Since J(-) is strictly convex, k{x) is unique at each x € Frojjf^. 

The constraint (x, u) ^ Z imposes an implicit state-dependent constraint 
u €U{x) on u where the set-valued function U{-) is defined by 

U{x) = {u \ {x,u) S Z}. 

We claim that ZY(-) is continuous (both outer and inner semi-continuous on 
X = Projjf^, the domain of By definition [23], the set- valued map 

is outer semi-continuous at x € A" if U{x) is closed and if, for any 
compact set G such that U{x) n G = there exists an e > such that 
U{x)r\G = for all x' € B{x, e)nX. The set-valued map U{-) is inner semi- 
continuous at X & X if, for any open set G C iR™ such that G PI U{x) / 0, 
there exists an e > such that G PI U{x') ^ for all x' G B{x,e) n X. 
Here B^{x,e) = {x' € \ \x' — x| < e}. The set-valued map is outer 
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semi-continuous because its graph, Z, is closed so that, given any sequence 
{{xi,Ui)} in Z {ui G U{xi) for all i) such that (xi,Ui) —> {x,u), we have 
{x,u) G Z so that u G U^x). Hence U{-) is outer semi-continuous [23]. 
We can establish inner semi-continuity using the following result [24] whose 
proof is given in the appendix. 

Lemma 1 (Clarke). Suppose Z is a polytope in iR" x iR™ and let X denote 
its projection on (X = {x \ 3u (z such that {x,u) E Z}). Let 
lA{x) = {u \ (x, u) E Z}. Then there exists a K > such that, for all x, x' G 
X , for all u E U{x), there exists a u' ^ U{x') such that \u' — u\ < K\x' — x\. 

Let X, x' be arbitrary points in X and lA{x) and lA{x') the associated sets 
(Figure ^ illustrates the proof for two cases: x = xi and x = X2). Let 
G be an open set such that U{x) r\ G 7^ and let u be an arbitrary 
point in U{x) fl G. Because G is open, there exist an e > such that 
B{u,e) = {v \ \v — u\ < e} C G. Let e' = e/K. From Lemma^ there exists 
a n' E U{x') Ci G for ah x' E B{x, e') n X. This implies U{x') n G / for all 
x' E B(x,£') n X, so that ZY(-) is inner semi-continuous. ■ 



To solve the parametric problem Pmim we develop further the reverse 
transformation procedures proposed in [25] and utilized in [4,7,8]. We as- 
sume that J(-) is strictly convex and continuous piecewise quadratic on a 
polytopic partition = {Pf' \ i E J^^} (for some index set J'^) of Z. For 



u 





B{x2,£'2)nX 



Figure 1: Inner semi-continuity of ZY(x) 
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each z = {x,u) € Z, let S^{z), the index set of active polytopes at z, be 
defined by 

S^{z) ^{ieJ^\zePf} (3.5) 

so that S^{z) is the set of indices of active polytopes at z. Similarly, for 
each X X = Proj^^ (2), let S^ix) be defined by 

SO^(x) ^ S^{x,k{x)) (3.6) 

where k{x), the solution of Pmin(a;), is defined by 

k{x) = argmin{ J(x, n) | {x,u) G Z} (3.7) 

u 

so that ^^(x) is the index set of polytopes active at the solution k{x) of 
^min{x)- For each i G J^^ we consider the simpler problem Pi(x) defined by 

Fi{x) : V^ix) = min{Ji(x,n) | {x,u) G Pf} (3.8) 

u 

Ki{x) = argmin{Jj(j;, u) | {x,u) G Pf} (3-9) 

u 

where J{z) = Ji{z) on P^ and 

Ji{z) = {l/2)z'QiZ + q[z + Si (3.10) 

for some Qi,qi,Si, all z = {x,u) G Pj^. For each i, problem Pi(x) is a 
quadratic program since J(-) is quadratic on the polytope Pf'. 

Proposition 2 Suppose J : Z ^ M is strictly convex and continuous and 
that Z is a polytope with a polytopic partition {P^ \ i G J'^}- Then, for 
all X G P^ = Projjj^Pj^, all i G J'^ , the solution Ki{x) to Pj(x) exists 
and is unique. Moreover, the value function V^{-) is strictly convex and 
continuous, and Ki{-) is continuous, in P^ . 

The proof of Proposition El is almost identical to the proof of Proposition 
n noting that, for each i G J""^, P^ is a polytope. Problem Pi(x), al- 
though simpler, is an artificial problem since the constraint (x, u) G P^ 
does not appear in the original problem Pmin(x), so that it is not obvious 
how best to relate the solutions to problems Pi(x), i G , to the solution 
of ^m\n{x)- This difficulty was not totally satisfactorily dealt with in the 
literature quoted above but is resolved in the following result. 

Proposition 3 Suppose J : Z ^ M is continuous piecewise quadratic and 
strictly convex on a polytopic partition of Z. Then u is optimal for the 
minimization problem Pmin(a^) if dnd only if u is optimal for the problems 
Pj(x) (i.e. if and only if k{x) = Ki{x)) for all i G S^ix). 



9 



Proof: Suppose u = k{x) is optimal for Pmin(x) but that, contrary to what 
is to be proven, there exists an i € S^ix) such that u is not optimal for Pi(rc). 
Let Ui = Ki{x) denote the solution of Pi(x). By definition, {x,Ui) G Pf' and 
{x,u) G (since {x,u) G P?' for all j G Hence Ui = Ki{x) satisfies 

V^'{x) = Ji{x,Ui) = J{x,Ui) < Ji{x,u) = J{x,u) = V^{x) where we have 
made use of the fact that J{x,v) = Ji{x,v) if {x,v) G P,f. Hence J{x,Ui) < 
V^{x) which contradicts the optimality of n for Pjnin(x). Suppose, next, that 
u is optimal for ¥i{x) for all i G S^ix) (so that Ki{x) = u for all i G ^^(a;)) 
but that, contrary to what is to be proved, u is not optimal for Fmmix) so 
that there there exists a u* satisfying (x,u*) G Z and J{x,u*) < J{x,u). 
Because u e Pf for all i G 5|(x) and d{u, Pf)>0 for ah j e \ 5|(x), 
there exists a A G (0, 1] and an i G S2;{x) such that ux = u+X{u*—u) satisfies 
(x,ma) G pf ■ Since u i— >■ J{x,u) is convex and J{x,u*) < J{x,u) it follows 
that J{x,ux) < J{x,u). But J{x,u\) = Ji{x,u\) (since {x,u\) G Pf) 
and J{x,u) = Ji{x,u) (since {x,u) G Pf ) so that Ji{x,ux) < Ji{x,u), a 
contradiction of the optimality of u for Pj(x) for all i G ^^(a;)- ■ 

Proposition 121 shows that the solution to Fmmix) is also the solution to a 
set of quadratic programs, namely Pi(x) for i G 5^(2;). We now analyse 
problem Pj(x) in more detail. Suppose that, for each i G Jl^, polytope Pf 
is defined by 

pf ^{z = {x,u)\M^u<NiX+p^} (3.11) 
where Mj, Ni,pi each have rows, so that 

Fi{x) : V^{x) = min{ Ji(x,n) | MiU < NiX + pi} 

u 

Ki{x) = arg min{ Ji(x, -u) [ MjU < NiX + Pi}- (3-12) 

u 

The j*^ constraint M^u < N-x + p\ is said to be active at (x,u) if M^u = 
N-x + pI- The set of active constraints for Pi(x) is lf{x), the set of con- 
straints active at {x,Ki{x)), so that 

I°(x) ^ {j G Ir, I M//^i(x) = iV/x (3.13) 

where the superscript j on a matrix (or vector) denotes the j^^ row of the 
matrix (or vector). It follows from the definition of Kij{x) = Kijx+kij that 
lf{x) = I for all X G interior{X}jj and that /°(x) C I on the boundary of 
Xij. The solution to Pj(x) is simple if the set of active constraints lf{x) for 
the problem is known in advance [3,26]. Suppose therefore the set of active 
constraints for Pi(x) at (x,Kj(x)) (Kj(x) is the solution of Pj(x)) is known. 
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apriori, to be /, i.e. I^{x) = I. Then Pi(x) is replaced by the simpler, 
equality constrained, problem 

Fij{x) : V°j{x) = min{ Jj(x, u) \ M^^u = Nfx + pj, j G /} (3.14) 

This is a quadratic optimization problem with affine equality constraints; 
the solution to this problem has, as is well known, the form 

VUli^) = {l/2)x'Q,jx + q[ix + Sij (3.15) 
Kij{x) = Kijx + kij (3.16) 

for some Qij, Sij, Kij and kij. Let M- denote the matrix with rows M- , 
j G /. Let' Pc'ij = {{Ml)'\ I A > 0} C iR™ denote the polar cone at 
to the cone Tij = {h \ Mfh < 0, j G /} of feasible directions h for 
problem 'Pi{x) &t u = Kij{x). The polar cone depends solely on /, the 
set of active constraints; it does not depend on the parameter x; also [4] 
—^uJi{x, Kij{x)) G PCij if and only if {VuJiix, Kij{x)), h) > for all 
feasible directions h [h ^ if and only if Kij{x) is optimal for the 

problem mmu{Ji{x,u) \ M-u = N^x+p!-, j G I^Mfu < Nix + p'^, j G 
Xr- \ /}. For each i G , let P^^ denote the polytope defined by 

P^^{x\ 3u s.t. {x,u) G P,^} = Proj^(/>^) (3.17) 

The polytope P^ is the domain of V^{-). The following result holds [4]: 

Proposition 4 The affine control law is optimal for problem ¥i{x), 

at all X in the polytope Xij C P^ defined by 



x^j^lxex 



M/ {Kijx + kij) < N^x+pi, j£ J,, \ / 

-VuJi{x,Kij{x)) G PCiJ 



(3.18) 



The restriction x G X is included in the definition of Xij since it is 
not included as a constraint in Pj(x)). Since the affine control law 
is such that the equality constraint MiKij{x) = NiX + pi is satisfied for all 
X, and since the last inequality in ()3.18l) ensures that Hij{x) is optimal for 
min„{Ji(x,u) | M/n = N^x+pj, j G /,M/n < N^x+p^, j G Ir, \ I}, it 
follows that Kij{x) is optimal for Pi(x) in the polytope Xij. Thus [3,4,26] 
the solution to Pi(x) is affine, and the value function quadratic, in each 
polytope Xij; the set of all such non-empty polytopes (as / ranges over 
the subsets of {1,2,... ,rj} constitute a polytopic partition of Fioix{Pt^) 
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so that the solution Ki(-) to is piecewise affine, and the value function 

V^{-) is piecewise quadratic, on this polytopic partition. 

However, in our case, since we have to 'marry' a set of polytopes Xij. 
for all i such that polytope Pf' is active (i.e. i G S^ix)), is active, it is 
preferable to parameterize the polytopes in which the solution to Pj(x) is 
affine by the state x, say, at which /j is active rather than by the set /j 
of active constraints. Also, for each x G = Proixi^), the polytope 
Xi{x) C Pf^ be defined by 

X,(x)^X,,,o(,) (3.19) 

where, for each set / C J'^ ^ Xij is defined by (|3.18|) . It follows from 
(|H.18|) . with / replaced by lf{x), that x E Xi{x). It was shown above that 
(x) = lf{x) for all X in the interior of Xi{x); it follows from 1)3.19(1 that 
Xi{x) = Xi{x) for all x in the interior of Xi{x). 

The polytope X{x) that figures in the parametric solution of Fmin(a^) is 
defined, for each x £ Af, by 

X{x)^f]{X,{x)\ieS%{x)} (3.20) 

Clearly x € X{x) and X{x') = X{x) for all x' E X{x). For each x € X, let 
the functions V^(-) and be defined on X{x) by 

V^{x)^V!^(x), yieS%{x) (3.21) 
Ks(x) ^ K,(x), Vi G 5i(x) (3.22) 

The domain of each function is X{x); that the functions are well defined 
follows from Proposition |21 and equations (|3.19() and (|3.2U() which show that 
Ki{x) = Kj{x) for all i,j G S^ix), all x G X{x). Summarizing, we have: 

Theorem 1 Suppose J : Z ^ IR is continuous piecewise quadratic and 
strictly convex on a polytopic partition = {Pf' \ i G J^^} of Z and that, 
for each i G , Pf" has a non-empty interior. Then the value function 
V^{-) is continuous piecewise quadratic and strictly convex on a polytopic 
partition = {P.^ \ i G J^} of X= Projj(^(-2). The minimizer k(-) is 
piecewise affine on . The polytopes P^ are each of the form X[x) for 
some X G X; the value function and optimal control law satisfy V^{x) = 
V^{x) and k{x) = Kx{x) for all x G X{x) and some x G X. 

The proof of this result follows from Propositions 01 and |1] and the discussion 
above. The result is illustrated in Figure El for the simple case when Z = 
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Vy U V2 has two partitions Vf and in each of which J(-) is quadratic. 
Problem Vf{x) is, therefore, a parametric quadratic program; its solution 
Ki(x) is known to be piecewise affine on a polytopic partition of \ in 
Figured = X and the polytopic partition is {Xn, X12, ^13}. The 
solution to Vf{x) is ni{x) which is affine in each of the polytopes Xn, X12 
and X13. Similarly the solution to the quadratic program Vf{x) is K2 (x) 
that is piecewise affine on a polytopic partition {X21 , X22 , X23 } of "P^ = X . 
The sets and the optimal control laws ki(-) and ^^2{^) are shown in the 
Figure. At each x ^ X, there are two candidates ki{x) and K2{x) for the 
optimal control k{x) for the original problem Pjnin(2;)- Theorem 1 resolves 
this difficulty; at all z on the boundary between and Pf , S^{z) = {1, 2} 
since both polytopes are active. Hence on the boundary, a control u is 
optimal for the original problem Pmin(a^) if and only if it is optimal for 
both problems Vi{x) and 'P^(x). At all x G Xn U X12 (except at its 
intersection with X13), only {x,K2{x)) lies on the boundary between Pf 
and P2' ((x, Ki(x)) does not lie on this boundary). Thus, in Xu U X12, the 
optimal control is not K2(x)); it is k(x) = ki(x). At all x GX13 n X21 both 
(x, Ki(x)) and (x, ^2(2;)) he on the boundary between Pf and so that the 
optimal control here is k{x) = ki(x) = K2{x). Finally, at all x G X22 U X23 
(except at its intersection with X21), only (x,ki(x)) lies on the boundary 
between Pf' and ((x,K2(x)) does not lie on this boundary); thus, in 
X22 U X23, the optimal control is k(x) = K2{x). Hence k(-) is completely 
defined on X. This procedure avoids the overlapping of sets that results in 
previous analysis of this problem. 

3.2 The maximization subproblem Pmax 

In the minimization subproblem the function V{-) being minimized 

is convex in both x and u. In contrast, in the maximization subproblem 
p.2|) . the function V{-) being maximized is convex in z and (under suitable 
conditions) concave in w. Hence we proceed somewhat differently. 

Proposition 5 Suppose V : ^ ^ M is such that z 1— > V{z, w) is strictly 
convex and continuous for each w G W , w 1— > V{z, w) is strictly concave 
and continuous for each z in Z = Proj^($), and that ^ is a polytope with 
a non-empty interior. Then, for all {z,w) G the solution v{z) to Pmax(-z) 
exists and is unique. Moreover, the value function J^{-) is strictly convex 
and continuous with domain Z, and u{-) is continuous in Z. 
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Figure 2: Problem P(x) 



Proof: Since J^{-) is the maximum of a set of strictly convex and con- 
tinuous functions, it is also strictly convex and continuous. The existence 
and uniqueness of ^{z) for each z € Z follows from the strict concavity and 
continuity oi w w) and the compactness of W. The continuity of 

u(-) follows from the uniqueness of u{z) at each z (Theorem 5.4.3 in [23]). ■ 

To obtain a parametric solution to Pmax) we assume that V{-) is continu- 
ous piecewise quadratic on a polytopic partition = {P* | i G ^7*} of 
^ = Z X W {in the absence of additional restrictions, Z = X x U so both 
Z and <I> are polytopic). For each {z,w) G let S^{z,w), the index set of 
active polytopes at {z,w), be defined by 

w)^{iej'^\ {z, w) G P,^} (3.23) 

so that S^{z, w) is the set of indices of active polytopes at {z, w). Similarly, 
for each z £ Z = Piojzi^), let S^{z) be defined by 

Si{z)^S'^iz,u{z)) (3.24) 

where 

^{z) = aTgmax{V {z,w) \ w G W} (3.25) 

w 

so that S^{z) is the index set of polytopes active at the solution i^{z) of 
Pinax(-2). For each i G ^7*, each z G Z, we define the simpler problem Pi(z) 
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defined by 

¥i{z) : J°(z) = max{V,{z,w) \ {z,w) G P,^} (3.26) 

w 

Vi{z) = argmax{;^,(z,u;) | {z,w) G P^] (3.27) 

w 

where V{z,w) = Vi{z,w) on and Vi{-) is quadratic. For each i, problem 
Pf (z) is a quadratic program. 

Proposition 6 Suppose V : ^ ^ M is strictly concave (hence continuous) 
in w for each z G Proj^^>, and that ^ is a polytope with a poly topic partition 
{P* I i G such that, for each i G J^^ , has a non-empty interior. 

Then, for all z G Pf' = Proj^P*, all i G the solution Vi{z) to Pi(z) 
exists and is unique. Moreover, the value function J|'(-) is strictly convex 
(hence continuous) with domain Pf' , and fj(-) is continuous at any z G Pf' . 

The proof of Proposition IHl is similar to the proof of Proposition IHI The 
relation between the solution to Pmax and the solutions to the subproblems 
Pf , i G is given in the next result. 

Proposition 7 Suppose V . ^ ^ M is continuous piecewise quadratic, 
strictly convex in z and strictly concave in w and is continuous piecewise 
quadratic in a polytopic partition P* = {P* | i G ^7*} of the polytope $. 
Then w is optimal for the maximization problem Pmax(-2) if and only if w is 
optimal for the problems Pi(z) {I'iz) = I'i^z)) for all i G S^{z). 

The proof of Proposition [7| is similar to the proof of Proposition |21 We now 
exploit the continuous piecewise quadratic nature ofV{-). For each i G ^7*, 
subproblem Fi{z) may be expressed as: 

Fi{z) : J^{z) = ma^{Vi{z,w) \ MiW < NiZ + pi] (3.28) 

{Vi{-) is quadratic) for some Mi,Ni,pi, each matrix (vector) having rows. 
If we assume that the constraints indexed by / C 2^.. are active, then Pi(z) 
is replaced by the simpler, equality constrained, problem 

Fij{z) : J°(z) = max{Vi{z, w) \ M^w = N^z + pj, j G /} (3.29) 

w 

where the superscript j on matrix (or vector) denotes the j^^ row of the 
matrix (or vector). The solution to this problem is 

4jiz) = {l/2)z'Qijz + qIiZ + s,,/ (3.30) 
i^ijiz) = Kijz + kij (3.31) 
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Let M/ denote the matrix the rows of which are , j S /. Let PCij = 
{(M/)'A I A > 0} C iRP denote the polar cone at to the cone J^ij = {h \ 
M^h < 0, j G /} of feasible directions h for problem Fi{z) at w = Vij{z); 
the polar cone depends solely on /, the set of active constraints; it does not 
depend on the parameter z. The following result holds [4]: 

Proposition 8 The affine control law is optimal for problem 'Pi{z) 

at all z in the polytope Zij defined by 



Zij = Iz e Z 



Ml[K,jz + hj) < Nfz + pI j elr,\I 



(3.32) 



(the restriction z ^ Z = X is included in the definition of Zij since it 
is not included as a constraint in Pi(z)). As before, we have to 'marry' a 
set of polytopes Zij for all i such that polytope P* is active. Suppose, for 
each i G J"*, polytope is defined by 



= {{z, w) I Miw < Niz + Pi] (3.33) 

for some Mj, Ni,pi each having rows. For each i G J^, let Pf' denote the 
polytope defined by 



Pf^{z\3w s.t. {z, w) G Pn = ProiziPn (3-34) 

For each z G Pf', each i G JT'*, the set of active constraints for ¥i{z) is 

/°(z) ^ {j G Ir, j M^Uiiz) = N^z+pi) (3.35) 

where is the j^^ row of Mi, the j^^ row of Ni, and p^ the j^^ row of 
Pi- Also, for each z & Z, let the polytope Zi{z) C P^-^' be defined by 

^iiz) = ZijO(^z) (3-36) 

where, for each index set / C J"*, Zj^/ is defined by (|3.32j) . The polytope 
Z{z) that fig 11168 in tliG pajrctiiiGtric solution of IPmax 

(z) is defined, for each 

z G Z, by 

Z{z) ^ n{Z,{z) I i G Sg(z)} (3.37) 

Clearly z G ^(^) and .Z'(z') = Z{z) for all z' G Z{z). For each z £ Z, let the 
functions J2(-) and l'2{^) be defined on .Z(^) by 

jO(z) A jO(^)^ vi G Sg(z) (3.38) 
i^s(^) = i^iiz), Vi G 4(z) (3.39) 
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The domain of each function is Z{z); that the functions are well defined 
follows from Proposition [3 (|3.35|) and H3.36() which show that t'i(z) = I'jiz) 
for all i,j G S^{z), all z E Z[z). Summarizing, we have: 

Theorem 2 Suppose V : ^ ^ IR is continuous piecewise quadratic and 
strictly convex on a polytopic partition = {P* | i G JT^*} of the poly- 
tope ^ and that, for each i G J'^ , has a non-empty interior. Then the 
value function J°(-) is continuous piecewise quadratic and strictly convex on 
a polytopic partition = {Pf \ i G J^^} of Z = {z \ 3ut G W s.t. {z, vu) G 
$} = Proj^(<I>). The maximizer z^(-) is piecewise affine on . The poly- 
topes Pf' are each of the form Z{z) for some z G Z; the value function and 
optimal control law satisfy J^{z) = J^{z) and v{z) = Vziz) for all z G Z{z) 
and some z ^ Z. 

The proof of this result follows from Propositions El and 0] and the discussion 
above. As stated above, in the absence of further restrictions, ^ = X x 
U xW . However, in our use of this result, the cost function V{-) has the 
form V{z,uj) = £(z,w) + V^{Fz + Gw) where F = [A, B] and V^{x) may 
be known only on a compact subset X of iR"; in this case ^> = {{z,w) G 
X X U X W \ Fz + Gw (z X}. If A is invertible (which we assume) or X is 
compact, then $ is a (compact) polytope with a polytopic partition. 

4 control; no state constraints 

In this section, we consider the H^o constrained optimal control problem 
when the only constraints are u ^ U and w G VF, i.e. X = Xf = iR". In 
this case, the dynamic programming equations (|2.1j) - (|2.4j) simplify and are 
replaced by the conventional dynamic programming equations: 



(x) = mmmax{£(x,u,w) ->r V^_i(f(x,u,w))} 



(4.1) 



Kj(x) = argmin max{^(x, u, ti;) -\-Vj_i{f{x,u,w))} 



(4.2) 



with boundary condition 
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rewritten in the form 



Vj>{x) 


= min J,? ^ix,u) 


(4.4) 




^xn^^U{z,w)+Vf_^{f{z,w))} 


(4.5) 


Kj{x) 


= arg min J^_^ (x, u) 
u£U ■' 


(4.6) 


Vj{z) 


= arg max {^(z, + Vf_i{f{z,w))} 


(4.7) 



where z = {x,u). The important feature of this problem is that the con- 
straint G [/ in subproblem (|4.4() has the same simple form as the constraint 
w G W; this permits us to obtain stronger properties for the value functions 
Vj^{-) and Jj{-), j > 0. The prototype problems for ()4.4|) and 1)4. 5|) are, 
respectively: 

min{J(x,u) \ u e U} (4.8) 

u 

max{V(z,w) I w G W} (4.9) 

w 

in which V^{-) and J(-) replace, respectively, ^^^(■) and Jj_i{-) in (|4.4j) 
and J^{-) and y(-) replace, respectively, Jj_-^^{-) and £(•) + V^_i(/(-)) in 
(|4.5j) . Since J(-) is convex in x and !/(•) is (under appropriate condi- 
tions) concave in w, their respective value functions have identical properties 
(min^jj J(x, m) \ u G U} = — max„{ — J(x, u) | u G U}). 

Proposition 9 Suppose that J(-) in Pmin (^(O IPmax) continuously 
dijferentiable, strictly convex in u (strictly concave inw) and that U (W ) is 
compact. Then the value function V^{-) o/Pmin (J^i') o/Pmax) is continu- 
ously differentiable. 

Proof: It is only necessary to consider Pmax and establish the continuous 
differentiability of J^{-)- Since W, being constant, is continuous in z, the 
continuity of the value function J^{-) follows from the maximum theorem 
(e.g. Theorem 5.4.1 in [23]). Since the function vu i— > V{z,w) is strictly 
concave for all z, the maximizer z^(z) is unique (a singleton) for each z; 
by the same maximum theorem, z^(-) is continuous. Since V{-) is continu- 
ously differentiable and W is compact, and the maximizer z^(z) is unique 
and continuous, it follows from the proof of Theorem 5.4.7 in [23] that the 
directional derivative of J''(-) satisfies 

dJ°{z; h) = {d/dz)V{z, v{z))h (4.10) 



Pmin(x) : V\x) = 

Pmax(^) : J\z) = 
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at any z, any direction h. Hence J^{-) is Gateau differentiable at any 
z G Z with Gateau derivative G{z) = {d/dz)V{z,i'{z)). Since G(-) is con- 
tinuous, J^{-) is continuously (Frechet) differentiable in Z with derivative 
{d/dz)J'^{z) = {d/dz)J{z,ij{z)) [27]. m 

Although the domain of the value functions V^^(') is M" and that of 
the value functions Jji') is iR"'"''™', we restrict attention in this section to 
polytopic subsets of these domains. With this caveat, we now show that 
there exists a 7 > such that V{-) in Pmax, which represents Vf-ii-) in 
(|4.5j) and therefore has the form V{z,w) = £{z,w) + V^{Fz + Gw) where 
F = [A, B], is strictly concave in w. 

Proposition 10 Let X be a polytope in JR"' containing the origin in its 
interior. Suppose V{-) is defined by V{z,w) = £{z,w) + V^{Fz + Gw) where 
V^{-) is continuously differentiable and continuous piecewise quadratic on a 
polyhedral partition = {P^ | i € J^} of X . Then V{-) is continuously 
differentiable and continuous piecewise quadratic on a polyhedral partition 
= {i^* I i e J*} of the polyhedron ^> = {{z,w) e M"- x U x W \ 
Fz + Gw £ X} and there exists a 7* > such that V{-) is strictly concave 
in w for each z in Z = Proi^^ and all 7 > 7*. 

Proof: The continuous differentiability ofV{-) follows from the continuous 
differentiability of £{■) and V^{ ). Take any two points wi,W2 in W. For 
all A G [0, 1], let wx = wi + \{w2 — wi), and, for each z G Z, let the real 
valued function 0(-) be defined on [0, 1] by (/)(A) = V{z,wx). Suppose that 
yO(x) = {l/2)x'QiX + q'^x + ri in P^^ (for each i G J^). Then 

Viz, w) = {l/2){Fz + Gw)'Qi{Fz + Gw) + q[{Fz + Gw) + + l{z, w) 
= -{l/2)w'{-i'^I - G'QiG)w + b[w + a 

on the polyhedron = {(z, w) G iR" x U x W \ Fz + Gw G P^^}, where 
hi and Cj depend on z. For any e > 0, there exists a 7* > such that 
Ci = 7^/ - G'QiG > el for all 7 > 7*, alH G J^'^. The function (/){■) is 
continuously differentiable and satisfies: 

cP{X) = -(1/2)(/i'C,/i)A2 + 6,A + a 

(|)'{\) = -{h'Gih)\ + b^ 

for all A G [0, 1] such that Fz + Gw\ G P*. Since (/>'(•) is continuous, </)'(■) is 
strictly decreasing if 7 > 7* . It follows, by a trivial modification to the proof 
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of Theorem 4.4 in [28], that cl){X) > 0(0) + A((/>(1) - (/)(0)) for all A G (0, 1) 
which establishes the strict concavity of and, hence, of u; i— > V{z,w) 
if 7 > 7*- That V{-) is piecewise quadratic on a polyhedral partition of 
follows, with minor amendments, from the proofs of Proposition |H1 and 
Theorem |2| ■ 

We can now establish the main result of this section, characterization of 
the solution to the constrained Hoc problem when X = Xf = iR". We 
characterize the value functions Vj' on polytopic subsets Xj of the true 
domain iR" by assuming that the terminal cost function V/(-) is known only 
in a polytopic subset Xq of iR". 

Theorem 3 Suppose Vf{-) is continuously differentiable, strictly convex, 
and continuous piecewise quadratic on a polytopic partition Vq of a poly- 
tope Xq C iR". Then, there exists a 7 > such that, for each j > 0, there 
exists a polyhedron Xj on which the value function Vj^(-) is continuously 
differentiable, strictly convex, and continuous piecewise quadratic on a poly- 
hedral partition of Xj, and the optimal control law Kj{-) is continuous 
and piecewise affine on the same polyhedral partition Vj^ of Xj . 

Proof: Suppose Vj'_-^^{■) is continuously differentiable, strictly convex, and 
continuous piecewise quadratic on a polyhedral partition Vj^_i of a polyhe- 
dron Xj^i if 7 > 7j-i. Then, by Proposition there exists a 7^ > 7j_i 
such that {z,w) 1-^ £{z,w) + Vj'_i{f{z,w)) is strictly concave in w, con- 
tinuously differentiable and continuous piecewise quadratic on a polyhe- 
dral partition Vj'_i of a polyhedron ^j^i = {iz,w) G iR" x U x W \ 
Fz + Gw £ Xj^i}. By Proposition 13 the value function J°„i(-) is then 
continuously differentiable and, by Theorem |21 Jj_i{-) is continuous piece- 
wise quadratic and strictly convex on a polyhedral partition "P^i of a poly- 
hedron Zj-i = Proiz^j~i (and the disturbance law i/j_i(-) is continuous 
and piecewise affine on the same polytopic partition). Then, by Proposition 
ini ^^(0 is continuously differentiable and, by Theorem ^j'(') strictly 
convex and continuous piecewise quadratic on a polyhedral partition Vj^ 
of a polyhedron Xj (and the optimal control law i'j-i{-) is continuous and 
piecewise affine on the same polyhedral partition). ■ 
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5 Hoo control; state and control constraints 

In this section, we consider the i^oo constrained optimal control problem 
when the constraints are u£U,wGW,x(zX and the terminal constraint 
xn S The dynamic programming solution of the Hod problem requires 
the repeated solution of the two prototype problems Fmin(x) and Pmax('Z) 
defined in p.lf) and ()3.2() which we rewrite in the form: 

min{J(x,u) I u G l^(x)} (5-1) 

u 

max{V{z,w) \ w G W} (5.2) 

w 

where the set-valued function U{-) is defined, for all x £ X = Projx(-Z^), by 

U{x) = {u\ {x,u) £ Z} (5.3) 

The presence of state constraints complicates the solution of the i^oo prob- 
lem considerably. The extra complexity arises in the solution of Pmin('^) 
since the control constraint u € U{x) is now dependent on the parameter 
X in contrast to the simple constraint u £ U when no state constraints are 
present. The dependency of the constraint on x can cause the gradient of 
the value function V^{-) in problem Pmm(ic) to be discontinuous even if the 
function J(-) being minimized is continuously differentiable; Proposition 9 
is no longer necessarily true for problem ¥^[^{x). However, there do exist 
conditions under which this result is true. 

5.1 Particular case 

Assume that J(-) in 1)3. 1|) is continuously differentiable and continuous piece- 
wise quadratic on a polytopic partition = {Pf ,i € J^} of Z. We show 
below, despite the fact that the constraint set lA{x) now depends on the 
parameter x, that the value function V^{-) for Pmin is, under certain further 
assumptions, continuously differentiable vn X = VioixiZ). To do this, we 
first consider, as in §3.1, the simpler problems Pi(2;), i G J'^ , defined by 
1)3. 8|) . For each z, problem Pj(2;) is a quadratic program, with a value func- 
tion V^'^(-) that is continuous piecewise quadratic on a polytopic partition of 
the polytope = Projjf (P^^), and may be written in the form 

Pi(x) : Vj^{x) = min{ Ji(x,ii) \x£Ui{x)} (5.4) 

u 

where 

Ui{x) = {u I (x, u) € Pf] = {u I M,u < NiX + Pi) (5.5) 



Pmin(x) : V\x) = 

Pmax(2) : J\z) = 
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It is known (see Proposition 0] and Theorem^ that the value function Vl^{-) 
is continuous piecewise quadratic, being quadratic on polytopes Xij, each 
polytope characterized by a set / C 1^,, of active constraints, where is 
the number of rows of Mj; the sets Xij, I C X^. (excluding sets with no 
interior) constitute a polytopic partition of the polytope . We require 
the following result which is proved in the appendix. 

Proposition 11 Suppose (i), Ji{-) is continuously differentiable, (ii) Pf 
has an interior, and, (Hi) for any two adjacent polytopes, Xij^ and Xij^ 
say, in the polytopic partition of Pi'- , either Ii C I2 or I2 Ii- Then, Vf{-) 
is continuously differentiable in P^- . 

We establish next the continuous differentiability of the value function V^{-) 

for Pmin. 

Theorem 4 Suppose that J(-) in P-H) is continuously differentiable and 
continuous piecewise quadratic on a polytopic partition = {Pf',i € J^^} 
of Z and that hypotheses (ii) and (Hi) of Proposition [771 are satisfied (hy- 
pothesis (i) is satisfied automatically) for each problem Pj, i E J'^ . Then 
V^(-), the value function for ¥min, is continuously differentiable in X. 



Proof: It follows from Theorem ^ that, for each i e , V^{x) = Vl'{x) 
for all X in the interior of each polytope Xij in the polytopic partition of P^'^. 
It follows from Proposition 1111 that V^{-) is continuously differentiable in 
P^" for each i € . Consider next a point x on the boundary between two 
polytopes P.f' and P^ ; clearly i and j both lie in S^{x) and, from Theorem 
[Tl yO(x) = V.l^{x) = Vf{x) for all x E X(x), so that V^{-) is continuously 
differentiable in X{x) and, hence, on all boundaries between polytopes in 
the polytopic partition oi X . ■ 



Theorem 5 Suppose Vy(-) is continuously differentiable, strictly convex, 
and continuous piecewise quadratic on a polytopic partition Vq of a polytope 
Xq C IRJ^ . Then, there exists a 7 > such that, for each j > 0, there exists 
a polyhedron Xj on which the value function Vj{-) is continuously differ- 
entiable, strictly convex, and continuous piecewise quadratic on a polytopic 
partition Vj^ of Xj, and the optimal control law Hj{-) is continuous and 
piecewise affine on the same polytopic partition Vj^ of Xj . 
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Proof: Suppose Vj^_i{-) is continuously differentiable, strictly convex, and 
continuous piecewise quadratic on a polytopic partition "Pj^i of a polytope 
Xj-i if 7 > 7j-i- Then, by Proposition [TUl there exists a 7^ > 7j_i such 
that {z,w) I— > £{z,w) + V^°_;^(/(2;, w)) is strictly concave in w, continuously 
differentiable and continuous piecewise quadratic on a polytopic partition 
Vf_^ of a polytope = {{z, w) e X x U x W \ Fz + Gw £ Xj_i}. By 
Proposition!^ the value function Jj-i{') is then continuously differentiable 
and, by Theorem [21 Jj_i{-) is continuous piecewise quadratic and strictly 
convex on a polytopic partition of a polytope -^j-i = Proj^^j-i (and 
the disturbance law is continuous and piecewise affine on the same 

polytopic partition). Then, by Proposition [^1 ^j^i') is continuously differen- 
tiable and, by Theorem ^ ^^(') strictly convex and continuous piecewise 
quadratic on a polytopic partition Vj^ of a polytope Xj (and the optimal 
control law ^'j_i(-) is continuous and piecewise affine on the same polytopic 
partition) . ■ 



5.2 General case 

A simple characterization for the solution of the Hoo problem with con- 
trol and state constraints does not appear possible when the simplifying 
assumption of §5.1 is not made. Without this assumption, the value func- 
tion V^{-) for the minimization problem Pmin is not necessarily continuously 
differentiable at the boundary between polytopes in the polytopic parti- 
tion of X. Consequently, the objective function V{-) (which has the form 
V = i + V^) in the maximization problem is not necessarily concave, no 
matter how large 7 is chosen. The resultant cost function J(-) in the mini- 
mization problem is then piecewise max-quadratic, i.e. it is continuous and 
equal to the maximum of a finite number of quadratics in each polytope in 
a polytopic partition of its domain. It does not appear possible to obtain a 
simple characterization for a problem with this structure. 

5.3 Illustrative example 

The partial value functions and optimal control laws can be computed by 
solving the max and min subproblems associated (for each subproblem) 
with each set of potentially active constraints and each set of potentially 
active polytopes. Since the number of these sets is combinatorial, a better 
procedure, employed in our computations, is to select a state-control pair 
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z = {x,u) in the max subproblem, determine the active constraint set J 
or the active index set of polytopes s, and then compute the corresponding 
disturbance law and the region Z in which this control law is optimal, 
using Theorem 2. The procedure is then repeated for a new value of z 
not lying in the union of the sets Z already computed. Once the max 
subproblem computations are complete, a similar procedure is applied to 
the min subproblem using Theorem 1. 

Our numerical example is optimal min-max control of a constrained sec- 
ond order system defined by: 



X 



' 1 


1 ■ 




" 0.5 " 





1 


X + 


1 



u + w 



(5.6) 



The state constraints are x £ X := {x \ \x\oo < 10}. The control constraint 
is u £ U := {u \ \u\ < 1}. The disturbance is bounded: w £ W := {w \ 
\'w\oo ^ 0.1} The path cost function is quadratic with Q = 101, R = 1 and 
7 = 100. The terminal cost V/(x) is quadratic {l/2)x' Pjx with 
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20.6143 
5.9244 



5.9244 
14.2329 



The terminal constraint set Xf is defined by the 4 inequalities: (—0.9849 — 
0.3155)x < 2.1526; (0.9489 0.3155)x < 2.1526; (0.4369 0.8995)x < 0.7079 
and (-0.4369 - 0.8995)x < 0.7079. 





(a) Regions of x space, A'^ = 1 



(b) Regions of x space, N = 2 



Figure 3: 2D Example 



The polytopic regions for = 1 and N = 2 are shown in Figure 01 At 
N = 2, the state space is partitioned into 17 polytopes in each of which 
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the optimal control law is piecewise affine; the state control space Z is 
partitioned into 5 polytopes in each of which the optimal disturbance law is 
piecewise affine. 

6 iifoo receding horizon control 
6.1 Introduction 

Since we make use, in this section, of the solution for infinite horizon, linear 
unconstrained problem, we assume, in the sequel, that (A, is stabiliz- 
able and that (C, B) has no zeros on the unit circle where Q = C'C. Since 
Q is assumed to be positive definite, (C, A) is detectable. These conditions, 
and the fact that R is assumed positive definite, ensure that the conditions 
assumed in [29], Appendix B, are satisfied for the full information case. 
Hence there exists a 7 > such that a positive definite solution Pf to the 
associated (generalized) Hoo algebraic Riccati equation for all 7 > 7 and 
associated optimal control and disturbance laws u = K^x and w = K^jX 
respectively. It is shown in [29] that, under these assumptions, the state 
matrices Af = A-ir BK^ and A(. = A-\- BK^ + GKyj are both stable. 

The terminal cost function Vf{-) for the constrained control problem 
is defined by 

Vf{x) = {l/2)\x\%. (6.1) 

and satisfies 

Vf{Acx) - Vf{x) +e{x,KuX,K^x) = 0. (6.2) 

The terminal constraint set ^/ is chosen to be a disturbance invariant set (if 
it exists) for the system x+ = AfX + Gw, Af = A + BK^- Any disturbance 
invariant set Xf satisfies 

/(x, KuX, W") C X/ V X G X/ (6.3) 

We assume that the set W is sufficiently small, and that 7 is sufficiently large, 
to ensure the existence of a disturbance invariant set Xf which satisfies 

Xf C X, KuXf C U, K^Xf C W. (6.4) 

That the last condition in 1)6. 4|) can be satisfied follows from the fact [29] 
that — > as 7 ^ 00. A suitable Xf may be computed as follows: if 
W = {w \ CyjW < Cw}, choose 7 such that X = {x \ C^K^x < c^} is 
reasonably large; clearly K^X C W. Next, choose Xf to be a disturbance 
invariant set for x"*" = AfX + Gw satisfying Xf CI X n X and K^Xf C U. 
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Since there does not exist a disturbance w that can steer the system 
outside Xf given an initial state in Xf, the optimal policy for in is 
w = K^x. The closed loop system x+ = A^x, = A + BKu + GK^, is 
exponentially stable and the controller u = K^x maintains the state in Xf if 
the initial state is in Xj. We observe that the solution of the infinite horizon 
constrained H^o problem (defined by H2.9|) with = oo) satisfies: 

yl{x) = yf{x), K^{x) = KuX,yxeXf (6.5) 

since, by H6.4() . the control constraints are satisfied everywhere in Xf so that 
the solutions of the constrained and unconstrained problems coincide. 



6.2 Hao control: control constraints 

Since this problem, as stated in §4, has no terminal constraint and since 
Vf{-) defined above is a local rather than a global Control Lyapunov Func- 
tion (V/(-) is valid in Xj), standard stability results [30,31] (that enforce 
the terminal constraint) cannot be employed. However, it is possible to de- 
termine a domain of attraction for the H^o controller characterized in §4. 
Consider the following dynamic programming recursion: 

Vf{x) = min max{£(x, u, t;;) +Vf_i{f{x^u,w))} (6-6) 



Kj{x) = argminmax{£(x,'u, ti;) + Vj_i{f{x,u,w))} (6-7) 

X* = {x I fix, ^,ix), W) C X*_^} (6.8) 

with boundary condition 

Vo\x) = Vf{x), Xl = Xf (6.9) 

This is identical to the recursion H4.4() - H4.6|) except for the inclusion of the 
recursion (|6.8|) that yields the sets X*, j > 0. Whereas the domain of the 
value function Vj^{-) is IR^ for all j > 0, the importance of the sets X* 
derives from Proposition 1121 below. 

Proposition 12 For every integer j > 0, every x G X* : 

V^{x) = V^{x) (6.10) 

Kj{x) = Kao{x) (6.11) 
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Proof: Suppose, for some integer j, = V^{-) on Then, by 

V^{x) = min max{-^(x, u, tt;) + V^{f{x,u,w))} = V^{x) 

for all X € X*. Since Vf{x) = V^{x) for all x £ Xq = Xj, the desired result 
follows by induction. ■ 

Hence, the solution to the finite horizon problem in §4 is also the solution 
to the infinite horizon problem (in the restricted sets Xj) provided the 
terminal cost Vj(-) is chosen as described above. A practical consequence 
of this result is that, in computing the value function Vj'{-) (and K,j{-)), it 
is only necessary to consider those states lying in Xj \ X*_i (since Vj'{x) = 
y/_i(x) = Voo{x) at all X G X*_^). 

Definition 4 A set X is robust control invariant for x^ = f{x, u, w) if, for 
every x ^ X , there exists a u & U such that f{x, u, W) C X . 

It follows from (|6.3|1 that the set ^/ is robust control invariant. 

Theorem 6 (i) The sets Xj are each robust control invariant and are non- 
decreasing (satisfy X* C X*^^ for all j > 0). (ii) For any N > the set 
Xf is finite-time attractive with a domain of attraction X^ for the closed- 
loop system x^ = f{x,Koo{x),w). (Hi) Suppose that, for some finite integer 
j < N, Xf lies in the interior of X* ; then Xf is robustly stable for the 
system x+ = f{x,Koo{x),w). 

Proof: (i) Assume X*_^ is robust control invariant. It follows from 1)6. 8j) 
that ^ so that X| is robust control invariant. Since ^/ is robust 

control invariant, the desired result follows by induction, (ii) By construc- 
tion, for any integer j, and state x G X* is robustly steered into X*_i 
by the admissible control Kj{x) = Koo{x). Hence any state x G X'^ is 
robustly steered into Xf in steps; the controller u = K^x then keeps 
the state in Xf, so that Xf is robustly finite-time attractive with a do- 
main of attraction X^ for the system x+ = f{x,Koo{x),w). (iii) For any 
X G M"' let = d{x,Xf) and for any infinite sequence {x{i)} in iR" let 
|{x(i)}|^ = supj>Q (i(x(i), Aj). From (ii), the controller Koo(') steers any 
X G Xj into Xf in no more than j steps and, thereafter, keeps the state in 
Xf. Hence, with x{i) = (j){i; x, tToo, {w{i)}), w = {vu{0),u){l), . . . , u){j — 1)}, 
let e:X*x ^Mhe defined by 

6»(x,w) = |x(-)|^ = max{(/)(i;x,7roo, w} [ i G {0, 1, . . . , j - 1}}. 
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The control law «;oo(') is continuous since it is equal to Kj{-) in X*; thus 9{-) is 
continuous and, hence, uniformly continuous in Xj x . Since 0(x, w) = 
for all X € Xj C X*, all w G , uniform continuity of 9(-) implies that, 
for all e > 0, there exists a 6 > such that 9{x,-w) < e {x{i) G B^{Xf) for 
all i > 0) for all (x,w) G X* x satisfying \x\^ < 5 {x ^ Be{Xf)). This 
establishes robust stability of Xj. ■ 

The disadvantage of this approach is that the sets X* are obviously subsets of 
]R^, the domain of the value functions Vfi-)] the sets X^ are not necessarily 
convex. 



6.3 Hao control: state and control constraints 

Consider the receding horizon controller u = Kj\f{x). If the terminal condi- 
tions and assumptions stated above in §6.1 are adopted, then 

CI: Xf is robust control invariant for x~^ = {A + BKu)x + Gw, Xj C X, 
KuXf C U, K^Xf C W. 

C2: inmu£U ^'^^^w<=w{[V f+i]{x,u,w) \ f{x,u,w) G Xf} < for all x G Xf. 

In C2, V f{x,u,w) = Vf{f{x,u,w)) — Vf{x). If the recursive dynamic pro- 
gramming equations in (|2.H) - (|2.3|) are employed, we obtain, by a minor 
modification of the results in [22], §3.3.1, the following results: 

(i:) Xi is robust control invariant for all i G {1, . . . , N} 
(ii:) Xn is robust invariant for x+ = f{x, K,j\f{x), w) 
(iii:) VPix) < Vl.ix) Vx G i G {1, . . . , N} 

(iv:) V^{x) <Vf{x)y X e Xf. 
(v:) The value function satisfies: 

[(V;^ +i)< {V^ - V^_,)]{f{x, kn{x),w) < 

for all {x, w) G X]\f x W. Property (iii) is the monotonicity property of 
the value function for the constrained, linear, uncertain system (ll.lj) with 
cost (|1.5() . Let Vn {x , TT , w) denote the cost if player u uses the control law 
u = Kj\f{x) and the adversary w uses an arbitrary admissible disturbance 
sequence w. Then, for any £2 disturbance sequence 
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N-1 

lV(x,7r,w) = J2 - (7V2)|«^«P + Vf{x{N)) 

oo 

= Ely(^)l'-(^V2)kWP<^i^(^) (6.12) 

i=0 

since w is not optimal; herey(z) = Hz{i), z{i) = {x{i),u{i)) = {x{i), KN{x{i)) 
and x{i) is the solution of due to initial state x, control strategy n and 
disturbance sequence w; we make use of the fact that kn{x) = Kf{x) for all 
x € Xf. It follows that 

oo oo 

J2 \yii)\' < (7V2) E I^WI' + ^^(^) (6.13) 

i=0 i=0 

which is the finite gain property. Next, if the disturbance is identically zero, 
[{V^ + l)< {V^ - V^^-i)](/(x, ^7v(x), 0)) < 

so that 

V^if{x,KN{x),0))-V^ix) < -i{x,KNix),0) < -il/2)x'Qx 

for all X S Xn so that the origin is exponentially stable with a region of 
attraction X^^f. Summarizing we have 

Theorem 7 The receding horizon controller u = K]\[{x) has the following 
properties. The controlled system has the finite £2 gain property ()6.13|) for 
every initial state in the interior of Xn and, if the disturbance is identically 
zero, the origin is exponentially stable with a region of attraction X^. 

If the disturbance satisfies w E W{z), where, as before, z = {x,u) and W is 
such that w G W{z) implies \w\ < S\z\, for some 6 > (this W models some 
parametric uncertainties), then 

i{z,w) = {l/2)\z\U' - (7V2)kP > (c/2)|zp > (c/2)lxp 

for all z, w, some c > 0, provided that 6 < (I/7). With this form of bounded 
disturbance, the origin is robustly, exponentially stable (the state converges 
to the origin exponentially fast despite the disturbance) if, of course, 6 < 
(I/7). 
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We note, in passing, that we can simphfy the dynamic programming 
recursion, as in §6.2, by replacing 1)2.3(1 by 

x; = xn{x\ fix, Kjix), w) c 

and the boundary conditions by 

Vo\x) = Vf{x), Xl = Xf 

However, in this case, (|2.1() remains a constrained optimization problem be- 
cause of the state constraint, so the advantage of using this formulation is 
not so clear cut. As before, X^ C Xj for each j which introduces conser- 
vatism. However, the sets X* are less complex than the corresponding sets 
X,. 

7 Conclusion 

We have shown (in §4) how the solution to the constrained problem may 
be characterized when the system is linear, the cost quadratic and the con- 
straints polytopic if no state and/or terminad constraints are present. This 
characterization required the solution to a parametric program in which 
the constraints are polytopic and the cost piecewise quadratic (rather than 
quadratic). A novel solution to this problem is presented in §3. A charac- 
terization of the solution to the constrained problem when state con- 
straints are present under special (and restrictive) conditions is presented in 
§5; characterization of the solution in the general case appears to be diffi- 
cult. Stability properties of the resultant i?oo controlled system are briefly 
discussed in S6. 



APPENDICES 

A Proof of Lemma [H, §3.1 

We restate Lemma ^ 

Lemma 1 (Clarke) Suppose Z is a polytope in M"" x iR™ and let X 

denote its projection on iR" (X = {x \ 3u (z SV^ such that {x,u) G 2}). 
Let U{x) = {u \ {x,u) € Z}. Then there exists a K > such that, for all 
x,x' G X, for all u € U{x), d{u,U{x')) < K\x' — x\ {there exists a u' G U{x') 
such that \u' — n[ < K\x' — x\). 
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Proof: The polytope Z is defined by 

Z = {^z = (x, u) I Mu < Nx + p, Lx < I}. 

where the second set of inequahties is introduced to ensure that no row of 
M is zero {Lx < I defines the right hand boundary of Z in Figure Let r 
denote the row-dimension of M, s the row dimension of L, let / = {1, . . . , r} 
and J = {!,..., s}. For all {x,u) G Z, U{x) = {u \ Mu < Nx + p} 
(since (x,n) G Z =^ Lx < I). For each x € X, U{x) is a polytope in 
iR™. For each {x,u) £ Z, let 'ip{x,u) = max{M*n — N'^x — | z e /} 
and let ip'^{x,u) = max{0,ip{x,u) where M*, denote the ith row, 

respectively, of M, N and p. Then, for all x G X, u ^ U{x) ((x,u) € Z) if 
and only if ip{z) = 0). Let I^{z) = {i G I | M'li — N^x — p^ = i/^iz)} index 
the (most) active constraints. The associated set of (most) active gradients 
(with respect to u) is {gi \ i E I^{z)} where, for each z, gi = (M*)'. Because 
U{x) is a polytope, for each x G A", the set of active gradients is positively 
linear independent (0 ^ co{gi{z) | i G I^{z)} for all z such that ^(z) > 0. 
The proximal subgradient of n i-^ "fpix^ u) is: 

(5p(x, u)V'(x, u) = co{gi I i € I^{z)}. 

and the directional derivative d'ip{z; h) of 'ip{-) at z in direction h is max{((7j, /i) 
i € /'^(z)}; the positive linear independence condition ensures that, at each 
z, there exists a direction h along which ^{z) can be decreased. In fact, 
there exists a 5 > such that, \i z G Z and ^(z) > (z Z), then Q G '5p(2;) 
implies |C| > 5. So, by Theorem 3.1 in [32], for all (x,u) G 2, all x' G A", 
d{u,U{x')) < ip{x',u)/6 < {c/6)\x — x'\ (c = maxjlA^-'l | j G J}) since 
ip{x',u) < Tp{x,u) + max{A^-'(x — x') | j G J} < c|x — x'|}. This proves the 
lemma with K = c/6. ■ 



B Continuous differentiability of the value func- 
tion of a parametric quadratic program 

See [22] and [8] for related results. We consider the standard parametric 
quadratic program: 

P(x) : y°(x) = min{y(x,u) | (x,ii) G Z}. (B.l) 

u 

where x G iR", u G iR' and Z is a polytope with a non-empty interior. We 
assume 
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Al: The cost function F(-) is strictly convex and continuously differentiable. 

The constraint {x, u) (z Z imposes an implicit constraint on u E U{x) where 
the set- valued function U{-) is defined by 

U{x) = {u I (x, u) e Z] = {u \ Mu < Nx + p] (B.2) 
so that P(x) may be written in the form 

P(x) : V^{x) = min{y (x, u)\u^ U{x)) (B.3) 

u 

The (unique) solution of for each x G A", is 

u^{x) = argmin{F(x,rt) | u G U{x)} (B.4) 

u 

The domain of u(-) and U{-), is the polytope 

X = {x\ U{x) ^ 0} = {x I 3 n G U{x)] = Projx(-Z) (B.5) 

Let p > I denote the number of rows of M,N and c. It is known that 
V^{-) is continuous picccwise quadratic and continuous and u^{-) is piece- 
wise affine and continuous, being quadratic and affine, respectively, in the 
polytopes Rj, I CIp that constitute a polytopic partition of X. Each region 
is characterized by a set of active constraints /, i.e. for all x E Rj: 

Miu^{x) = Nix + ci (B.6) 
MiU < NiX + Pi for all i e I" (B.7) 
-VuV{x,u^{x)) G PCi{x) (B.8) 

where Mj, Nj and cj denote the matrices with, rows Mj, iVj and Cj, respec- 
tively, i e I, and PCi{x) = {M|A | A > 0} is the polar cone to the cone 
F{x) = {h I Mjh < 0} of feasible directions at x; for each I C 2p, P denotes 
the complement of / in Ip. Thus V^{-) is continuously differentiable (in fact 
analytic) in the interior of each region Rj, I C Ip. We may assume, without 
loss of generality, that Mj has maximal rank. Our final assumption is: 

A2: For any two adjacent regions Rj^ and Rj^ {Rj^ (iRj^ / 0) either Ii C h 
or Ii D h- 

Assumption A2 will often be satisfied, but there do exist counterexamples. 

Theorem 8 Suppose V{-) is continuously differentiable and that assump- 
tions Al - A2 are satisfied. Then V^{-) is continuously differentiable in 
X. 
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Proof: It is known that V^{-) is continuous piecewise quadratic and con- 
tinuous and u^{-) is piecewise affine and continuous, being quadratic and 
affine, respectively, in the polytopes Ri, I C Ip that constitute a polytopic 
partition of X. Each region is characterized by a set of active constraints I, 
i.e. Rj is defined by the inequahties HB.6() - HB.8() . Thus V^{-) is continuously 
differentiable (in fact analytic) in the interior of each region Rj, I C Ip. 
Consider the continuous differentiability oi V^{-) on the boundary between 
two regions Rj^ and R12 say where Ii C I2. For any I C. Ip such that 
Ri 7^ 0, any x G Rj, 

=Ci?(x) + u?(x) (B.9) 

where, for each index set /, \jPj{x) € range(iVfj)= M{Mi)^ (the row space 
of M/) is that u of minimum norm satisfying Mju = Njx + cj and v!j{x) G 
range (M|)^ = M{Mi) {N{Mi) is the null space of M/). Roughly speaking, 
\}Il\{x) satisfies the constraints, and u5(x) optimizes. It is easily shown that 
both u/(-) and u5(-) are affine in x, satisfying, respectively 

u?(x) = K/X + A;/, u?(x) = i?/x + fc/ (B.IO) 

where ki and the columns of Kj lie in range(Mj) and kj and the columns 
of Ki lie in range(Mj)^. In fact, Ki and ki are given by 

Ki = m\Ni, ki = m\ci (B.ll) 

where m|, the Moore- Penrose pseudo inverse of M/, is given by 

m\ = {MjMi)-^Mi (B.12) 

Since V^{-) is continuously differentiable in each region Rj, consider the 
continuous differentiability of V^'^(-) on the boundary between two regions, 
Ri^ and Rj^ say, where Ji C I2 Because, for x G i?/, u^{x) minimizes 
(with respect to u) the continuously differentiable function V{x, u) in the 
hyperplane {u \ Mf^u = N^-^x + c/^}, we have 

-V„y(x,u°(x)) G {A^A I A > 0} C range(M;j (B.13) 

Hence 

{d/dx)V^{x) = {d/dx)V{x,u^{x)) + {d/du)V{x,u°{x)){d/dx)u°j^{x) 

(B.14) 

since {d/du)V{{x,u^{x))){d/dx)u^j^{x) = (because of (|B.13() and the fact 
that {d/dx)u^_^{x) = KjJ and the columns of Kj-^ lie in range(Mj^)-'-. Sup- 
pose now X ^ X* G n R12 , X G Rj^ . Then 

{d/dx)V°{x) ^ {d/dx)V{x*,u'^{x*)) + id/du)V{x,u'^{x*))Kj, (B.15) 
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where both VuV{x,u^{x*)y and the columns of Kj-^ = {d/dx)u^^{x*) lie in 
range(MU. Next consider a x £ Rj^ such that x ^ x* G Rj^nRj^. Arguing 
as above we deduce 



{d/dx)V^{x) ^ {d/dx)V{x*,u''{x*)) + {d/du)Vu{{x, u'\x*))Ki^ (B.16) 

where S/uV{x,u^{x*)) lies in range(M|^) (as above) but the columns of 
Kj^ = {d / dx)\jLj^{x*) lie in range(Mj^). We show below that Ii C I2 implies 

that Mi^ki^ = Mi^Ki^. Since V„y((a;, u°(x*)) lies in range(M}J, it follows 
that 



{d/du)V{x,u^{x*))Kj^ = {d/du)Vu{{x,u^'{x*))Ki. 



(B.17) 



Equations ()B.15|) - ()B.17|) establish the continuous differentiability of V^{-) 
at X* e Ri^ nRi^. 

We have now to show that Ii C I2 implies that Mj-^Kj-^ = Mj-^Kj^. 
Suppose 



Mi, 



Mi, 
m 



Ni, 



n 



Then, from (IbTTIi 

But 
so that 



Mi.Ki, = Mi.mINi, 
Mi.Ki, = Ni, 
Ki,-- 



Mi, 
m 



n 



from which it follows that 

Mi.ki, = Ni, = Mi.Ki,. 
It follows from A2 that V'''(-) is continuously differentiable in X. 
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